Add lbzip2 2.6.5 - #322
Open
caius72 wants to merge 1 commit into
Open
Conversation
lbzip2 is a bzip2-compatible compressor whose command-line tool splits the work over threads. lzbench benchmarks buffer-to-buffer calls and does its own threading, so the wrapper drives lbzip2's low-level encoder and decoder sequentially, block by block, the way compress.c and expand.c do around their scheduler. Only the codec sources are vendored (crctab.c, decode.c, divbwt.c, encode.c, parse.c), unmodified; the tool's threading and I/O machinery is not built. Two -D renames keep the link clear of zstd's divbwt() and of any other xmalloc(), and a small arpa/inet.h shim covers MinGW, which lacks that header. The output is bzip2: bzip2 -t accepts it, and on silesia.tar the compressed size is within 0.005% of bzip2's at -9.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lbzip2 is a bzip2-compatible compressor. This adds it as
lbzip2, levels 1-9, benchmarked single-threaded like the other codecs.What is vendored
bwt/lbzip2/holds five sources from lbzip2 2.6.5 —crctab.c,decode.c,divbwt.c,encode.c,parse.c— plus their headers,COPYING(GPL-3.0-or-later) and aREADME.mdrecording provenance and how to refresh them. They are byte-for-byte upstream. Upstream'smain.c,process.c,compress.candexpand.care the multi-threaded command-line tool and are not built.lbzip2_lzbench.cis the lzbench-side wrapper. lbzip2's tool splits work over threads; lzbench benchmarks buffer-to-buffer calls and does its own threading, so the wrapper drives the same low-level codec sequentially, block by block, the waycompress.candexpand.cdo around lbzip2's scheduler. It also supplies the one function (xmalloc) thatdecode.cexpects from the tool. RegisteredBENCH_POOL_MT.Build and portability
-Ddivbwt=lbzip2_divbwt: zstd's dictBuilder exports adivbwt()too, which collides at link time.-Dxmalloc=lbzip2_xmallocpre-empts the same problem with a much more common name. Renaming on the compile line keeps the vendored sources unpatched.bwt/lbzip2/arpa/inet.his a shim for MinGW, which has no such header; three of the sources include it forntohl/htonlalone. It is reachable only through-Ibwt/lbzip2and defers to the libc macros wherever they already exist.vqtbl1q_u8under__aarch64__ && __ARM_NEON,_mm_shuffle_epi8under__SSSE3__, scalar otherwise. So aarch64 gets the SIMD path automatically, while a default x86-64 build gets the scalar one unlessMOREFLAGS=-march=native(or-mssse3) is passed — the numbers below are from an aarch64 machine, where the SIMD path is live.make DONT_BUILD_LBZIP2=1disables it; no CI leg needs excluding.Correctness
bzip2 -taccepts the output and roundtrip is exact on silesia.tar. Checked with no lzbench errors across all 9 levels,-b1and-b64chunking,-T4, incompressible random data, and 1- and 10-byte inputs.Numbers
Apple M5 Max, macOS 26.5, clang 17,
lzbench -ebzip2/lbzip2 -t8,8 -o1c4 silesia.tar. Sorted by compressed size, so each level sits next to its bzip2 counterpart:Compression is 1.59-1.73x bzip2's and roughly flat across levels, where bzip2 decays from 27.9 to 25.5 MB/s. Decompression is 1.34-1.52x. Compressed size is equal or marginally smaller: 0.004% at -9, 0.19% at -1.
Compression figures are one iteration per level (a single pass over 212 MB already exceeds the 8 s target); decompression got 2-5 iterations, stable to within ~1%.